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aBSTBACT 

As a part of the two-year process of revising the 
basic oatbematics course at the Open University CBritatii) , an 
in-depth survey of students completing the first unit of the course 
was confludted. i sample of 12C students was divided into throe groups 
of '♦0; group members received a questionnaire concerning concapts 
covered^ a questionnaire about sections of the unit, or a test on the 
Baterial. Approxiina tely half of the students in each group returned 
the questionnaires. The concept guestionnaire asked students to rate 
their prior faniliarity, effort needed to understand, and current 
understanding of each concept identified in a conceptual analysis of 
the unit. These ratings were subnitted to a multivariate analysis of 
variance. The sections of the unit were rated, and data analyzea 
similarly. Several relationships vere uncovered! difiieulty of 
concepts ^as predictea by amount of effort, and the time needed per 
section of a unit was related to level of concepts, nunber of 
diagrans, and nunbei- of lines in the text. The rating instruments and 
sunniaries of responses are included, (SD) 
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The D-n tailed Evaluation of MathomdticB Courses at the Open Unjv;^ 1-1 ity 
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'^'^ ^ MathDmatic*3 Foundation Course 



by 

Go r Jon J . Burt 



Detail edl Evrluatioi? irvolves obtsining InfuraatiDn fi'om 
students on t/*any detic^iled aspects the learning 
mattirlals^ analysing the content of the materials, and 
carrying out a atatiatical analysis of both the content 
and ciiG student-based infonnation^ This muthDd im 
applied to the first unit of the Matheinatics Foundatlnii 
Cpursa . 
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Jmroduct^ion 

Thi' lU^tailutl evaluation unit 1 iwoived afsking studGntB 158 quastions 
about various asppctsi ox' unit 1* A gystartfatic enalyaia of the quantita\;ive 
characfcc^rlsties of thn text and of its coricepttial structure has baen carried 
out* Multiple reqrassion analysl.^ ha.'i been uned to establlah the reXatiotiships 
which exist botwetin tho students' rating^ and the Dont^nt variables* The 
hierarchical 1 eyel of a concept or 0% a Sc?ctioii proves to have a powerful 
reLition?^hip with the f^tudGfits* mting^s 

Other results of the ovabjation include path diagrams illustrating 
studtnitf? ^ Btudy patterns, and the atudentg^ comamitB on the questionnaires 

1 ^ methocl 

Three quGStioniiai res were designed asking stucient^ about the conceptSf 
aectioua and assessment questions^ respective! for unit 1 of HlOOi (aee 
Appc?ndix l)* Three similar letters were deaigfied to go with each ciuestionnair© 
(see Appendix 3)* A sampla of 120 F--yeai, MlOO, students Vfas split into three 
ciroupB each with 40 students. The first, second and third groups received 
copiea of the cQncepts, sectioiis^ or assesamen^ questionnaire respectively. 
Table 1 illustrate& the time-tabl© associated ^Ith all three questlonnslrea^ 
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Ma roil IJ) & 



0Gnt "iues-:j omnaires 

i r-Gconi tended st^rt to study ) 

received return.^ 
anal y'<^ r estil is 

Matthemati cs Faculty 
wrote Report 



riffceivGd unit 1 

gCiw rvo 

TVl 
due date 

cut-off date 



rw^ceiVf u sunimary 



All throi^ quest Aonnai produced a r^e3pA)nsa from 31 out 0 ^ thu -tO 
s t uC n t B ^ Qu m t L o n n a x r '.^ s started com i ng ba ck almost jnirte d 1 n e 1 * Ct I'fTiph i 
plots the numbtr of sturtents v^ho had riSapcnded ovei a aisyen^weftl^i period. 




0 CdfK^t^. 



Fig 
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2, ;4 1 LKi fit 1 1 &jj_£yHU;iioiia to the qufts tionnalrea 

Ou two of the quGstionnaireEi I asked the studont if the instiiictionB 
wre ciioar what difficulti^a ho had conpleting the quastionnal re 

tha timo ho spent on it. TabU^ 2 gives the number of students saying thu 
LnKtructiona uero clerir, the number v^ho indicatod difficulty^ and the 
mf>dian arui range of timt/s spent* Appendix 3 gives the verbatim comnients 
of atudnntR, Completing a questionnaire is not an easy task, and I have 
rcdf^signod later ciuestionnaires, giving fuller inetiiJctionB wliieh meet 
somo of thn difficultieB mentioned. 



Tab] n 2 





Instruct ions 


Difficulty 


Median 




Qa i re.' 


clear? 


filling it in? 


time 


Ranje 


concepts 


Yes 22/22 




ji/m 


20 min 


7-45 






No ^ 


11/22 


10 min 


a-15 




Vppi . 17/21 


Yen - 


1 3/21 


1 5 min 


5-35 






No r:^ 


0/21 


10 min 


3-30 



4d v/ell as providing detailed infoimation on '♦'he assessment questions, 
the assesament queBtionnaire provided some infonnation on atndy patterns. 
Diagram 1 below gives the students* anewere to the questions ^'when did you 
3tart studying unit 17" These atudanta were new (F--year) students and so 
received the materials in early Decamber* The main point tn notice is 
that the starting date varied from 8th December to 3rd February. In all 
cases the actual starting date was earlier than the recommended starting 
date and earlier than the transmission of TVl . 
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Diapram 2 below giv€3S the time sijent by the students on each of the 
componentfl. The total time of 13 hours was apli between text (8^), 
/iBSOSsmont (a), tutorials (li), and broadcasts (l). Theaa figures should 
h(i taken to indicate rough orders of mannitude, Furthemiore, in many study 
sHituationa, it must have bean difficult for the student to decide whethar 
to include th'~^ study time in the 'text* nategory or in the 'asaesament* 
Cft te tin ry * 



Diaqram 




ffici? 



Table 3 shows the number of students who first read the (MA (/T>iA) 
qUGstions at various stages of their study. The patterti is the same for 
both CMA and TMA qusstions* The most common time to look at the questions 
is after the first reading of the text. However a good number of students 
road the questions before this stage. 

Table 3 The number of students who firat read the (OdA/TMA) questions I 



before any 


during first 


after first 


during later 


after later 


study 


reading 


reading 


readinf s 


readings 


3 


4 


H 


Z 


1 


3 


4 


11 


2 


1 
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auU ^ihowti the nunibor of studonts who :::ir3t attempted thu CMaC/TMA) 
f|iM Uions at variolic s^iagcB of their study. The pntt^rn is similar for 
bohli CMA and TMA questions. The moet coirunon time to first attempt the 
qLf:*'MianH Ly aQ^iin aftor tho firBt reading. Very few atudents attempt th« 
<|iip?ition^ boforr' thia stage* 

TMili^ k The ntimber of -students who attempt the (aiA/TMA) questions J 





bt*ft>rn any 
Htudy 


tluririg f i rst 
rf^ading 


after first 
readinfl 


during later 
readings 


after later 
readings 


CMA 




1 


11 


4 


5 


iMA 


1 


1 


9 


5 


5 



Tablf^ Bhowfi the number of Btudenta who completed the CMA(/TMA) question 
a; -j--viKun. stnno.^ in ihnlr i^tvidy. Again the pattern ±e similar for both CMA 
and TMA qunstions. The moat common time to coraplete th© questions is after 
snbHt^^quont readings - although some students manage to complete the questions 
/ifter idle first reading* 

table 5 The number of students who attempt the (CHA/TMA) questions J 





after 


fi rs t 


: reading 


during later readings 


af taJ 


^ later readings 


CMA 




5 




7 




9 


TMA 




6 




5 




10 



The above tables represent slices through the students' study patterns^ 
Diagram 3 illustrates the patterns themselves for the CHA quaations* The 
numbers in the circles are the numbers of studants who fDllowad that pattern* 
So, for example, 3 students first read the OTA questions iJefora any study (A) 
of the unit of these 3, 1 attempted the questions during his first 

readinci (S) and completad the questions after his firgt reading (C) of 
the other 2, they both attempted the questions after their first raading (C_) | 
and 1 of them complated the questions after hie firat raading (C) while the 
other 1 completed the questions dnring subsequent readingg (D)* 

Most of the students posted their assignraents batwaan 10th February and 
1st March* The due data was 27th February and the cut-off dote vras the 12th 
March, Graphs 2 and 3 show the number of students who had already posted 
their assignments by various dates through February • Tha two graphs show 
this for ™as and OTAs respectively. The two curves ara at their steepest 
(ie* the volume of returns per day is greatast) over the period ijrd to 27th 
February - the latter being the due date. It is surprising that students 
appear to be working to the due date, when, in fact, it is the cut-off date 
which is important - perhaps this is something these new atudents have still 
to learn. 



Befofff, mil 
re&i <|ns. 



mad 



4n| ff^t 



Mter fifst 



Pmm M€r I After ki^r 





completed 
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4, conce pt and sectiQn ratiiiga 



Table 6 ^ivas the mean ratings (averaged over the students) for epch 
concept in tha gloasary of Unit 1* The three quastions werei 

(1) How familiar were you with the concept before you studied the unit? (fthf) 

(2) How much effort waa required to gain your current understanding of 
this concept? 

(3) How good ia your currant undergtanding of the concept? 

The concepts are listed in the order in which they are presented in the 
unit. Of the first twelve concepts^ 'mapping' and 'function' dwanded tha 
moat effort from the student, and 'function' was the least understood deapite 
the fact that it was relatively fMiillar to the students beforehand^ A study 
of the text showed that the concept of •funotlon' was defined irtthout any 
supporting discussion or asCMples* 

Equality of mappings demands much effort and is poorly understood* The 
discuBslon of this concept in the teaet scots rather oonfUsed and no zeroise 
is set on it* The definition of dijumy variable is slipped Into the solution 
of an exercise - hence the low understanding rating. 

The concept of graph has the greatest prior familiarity, the mecoM 
lowest effort, and the greatest understanding* This fwnlllarlty eould well 
have been capitalised on to teaeh sow© of the earlier concepts* 
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The coru;c-i)t'4 of 'function of 2 variables^' and ''operator'* ciGmand a 
rvlaliva\y hinh arnoimt of effort and are relativfily poorly understood- 
Noithfir of thnso concopts are ess^sntial to the aims of the unit- The 
i:?.iUnijr>t.M wliicii aro mentioned in the objoctives are QstGriaked in tablG 1, 

The* ai'i thinnfcicnl cornbination of functions ( + , x, ^) caused few 
p rc i b I Of 11 ^ I Io w V G r t h e quo t i en t of t kc f u net i o ns requ i red ra t h e r more e f f o rt 
and wagi Igbb well understood with this concept one needs to be careful 
about df* fining tfe domain. 

Tht' rtiiiiaininy concepts demanded much effort and were pooi'ly 'understood , 
Simpler examples might aid the understanding of composition of functions. 
None of the examples on reverse mappings contained real functions and there 
wurr' no exerciaes at all. The diagram for classifi cation of mappings is 
confUr^ing* Finally, inverse function depends on the student undarstanding 
r eve rs a mappi ng . 



Tal 


c> 6 CQncei 


3ts, 


Unit 1 








prior 


effort' 


u nd e r g t a n d i n q ' 


1 


1 

8 tit 




27 


19 


43 


n 


el ement 






16 


46 


3 


equa I i t y o f sots 




26 


22 


43 




subset 




25 


22 


44 


5 


proper Butaset 




17 


23 


43 


6 


mapping 




23 




44 


t 


domain 




23 


21 


44 


a 


CO domain 




21 


23 


44 


9 


image 




23 


22 


44 


1 0 


function 




30 


28 


42 


11 


ordered pair 




39 


23 


43 


12 


variabl e 




34 


22 


44 


13 


equality of mappings 




16 


31 


39 


1 4 


dummy variable 




16 


25 


39 


15 


constant function 




33 


26 


44 


16 


g raph 




44 


17 


47 


17 


function of 2 variables 




36 


29 


41 


18 


operator 




18 


35 


37 


19 


di f ference 




39 


22 


45 


ao 


sum 




39 


22 


46 


21 


product 




39 


22 


46 


22 


quotient 




38 


26 


44 


23 


composition of functions 




ai 


34 


40 


24 


reverse mapping 




18 


33 


39 


25 


one^one mapping 




19 


28 


42 


26 


one^many mapping 




U7 


31 


41 


27 


many'^one mapping 




17 


33 


40 


28 


many-many mapping 




17 


31 


41 




inverse function 




19 


36 


40 



10 ^cmti psWj 9^ 0miHid 



TablQ 7 yivcH tiici mean rating for understanding and the mean time for 
oach subHOction of the tajtt. This infomotion confima the above discussion 
of tliP conc< nt rating© • 



tabl e 7 

Sections, Unit 1 



Unit ; 


t i fti© 


understanding i 










1 () 








1 a 


Map* function 






I. . 2 


(r.ombino fn. 






1 1 1 ) 


t nt rOe 






X 1 A 




11 


44 




^ ^^^^ 


13 


45 


113 


Mel 


l8 


4a 


1 1 H 


More det^ 


21 


42 


lis 


Specify nisp^ 


35 


42 


il6 


Graph 


32 


40 


11? 


Morn dc^f^ 


32 


43 


120 


Intro^ 






121 


Arithmetic 


14 


45 


122 


Coniposit ion 


29 


38 


123 


Decomposition 


15 


41 


624 


Reverse 


22 


39 


%25 


Classiry 


33 


39 


8 26 


Invei^se 


30 


36 


127 


Inverse CompOBite 


24 


39 
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_^he concep tual a X 



Each concept is defined in the gloasary. With tho exception of the 
definition of aot^ these definitions refer to other concepta in the gloeaary. 
The levoi of a concept is taken to be one higher than the level' of tho 
higheat^lGvel concept in th€' definition* 

Exanipl e * 



BEY: the definition does not contain any unit 1 













LEVEL 


I 


ELmmi I 


the definition 


mentions SET 


( level 


1) 


LETCL 


3 


SUBSET: 


the definition 


mentions SET 


(level 


1 ) and 










ELEMENT 


{ level 


2) 


LEVEL 


3 


EQUAL SETS: 


the definition 


mentions SET 


(level 


1) and 










ELmENT 


(level 


2) 


LEVEL 


3 


PROPER SUBSET^ 


the definition 


mentions SOT 


(level 












EL^NT 


( 1 evel 


2), 










SimSET 


( 1 evel 


3), 










EQUAL SETS 


( 1 evel 


3) 


^ • * IMVhL 


k 



Although the analysis does not require this, ve may illustrate the 
hierarchical level 3 of concepta in a diagrain. Diagram k shows this for the 
example diagram S for the entire unit* 



W^aQ^am 4 



sm 



LEVEL 1 



SUBSOT 



EL©^EOT 



EQUAL SETS 



=1 r 



PTOPER SUBSET 



LEVEL 2 
LEVEL 3 
LEVEL 4 



IS 



set 
s 



sulsft 

-1l 



tf ill' 



in 



11101 



1 



h — \ 



"~l I 1 

4 



ene-en< OiH-mMV (*i«y-»<»» (ii«V-"»«y 

. I 



I 

1 
3 

4 

ST 

6 
1 

B 
9 



6^ 


the multivariate amlyals of aoncept ratings 




Regregsion am 




BB ware earried out on ftva variables: 




LEVEL 


the 


I aval of the eoneept in the hiararehy (iee Diagram 5)i 




rnEQUENCY 


the 


nunibar of times the conaept was mentioned in tha tmt^ 


3^ 


PRIOR 


the 


mean rating for each conoept on 'prior famillarltyS 


4^ 




the 


niean rating for eaeh wncept an 'effort'. 


5* 




the 


mean rating for eadi eoaaept &n 'ourrant ufiderfltandlngv* 



First of all I used variables 1, 2 and 3 to pr^diot the effort 
demanded toy a ooncept* Diagrami 6 shovra the Gorrelstions between the thraa 
predictor variables . the variable 'prior familiarity* entsrr the ragresaioh 
first with a corr«latlon of 0.68 In m 29)* il«t, •level* mini's the 
regression and increases the muitlpl© correl&tloti t» 0.8l# The variable 
Hfreq^uenoy" does not inereaae the tt«ltlpi© corralation slgnifltontly* The 
regression aquation is therefore t 



EFTOW - 3 - 0,45 PRIOR + 0A2 IMWL 
B * .81 t ^ 4,5 5s 3.8 
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Dlagrani 6 

-68 

PRIOR - EFrORT 




PREQUENCy 



The residuals are interestingi 'operation'f 'eonpdsitlon' | ' fUnotlon' | 
and 'inveraa' require mueh more effort than predicted 'dunsn^ variable' ^ 

'prpper subsets 'image'f 'domain*! *nd the arltlunetlc of functlona require 
much less effort than predieted. These residuals suggest the urtneaeured 
effects of presentation , inherent diffloulty (given congeytual lev^l )^ and 
nay jertqigglogy » 

The next ragresalon used variables 2 and 3 to predlot the 'current 
understanding* of each corieept* In factf only 'prior fMlliarity' was 
significant, and the equation was^ 

UNDERSTANDING ^ 3*6 + 0.29 PRIOR 

The nm%t regression used variables 2| 3 ^nd 4 to yredlct the ^eurrent 
understanding' of each concept, 'Effort' had the highMt aorralatian (#87) 
with understanding « Next to enter the regressian WM 'level* incrtaaing 
the ntultiple correlation to OtBOm The equatioria werai 



UNDERSTATING ^ 5^3 - 0,49 OTFOKD + 0,0^ LENTIL 

1 ^ T 

R ^ .90 T ^ 9*5 t m a.5 



Studying the above three aquations 1 it sens clear that EtFOfO can bo 
regarded as a tneasure of eonoepttial difficulty hWOB the paradoigiDal 
result that greater effort leads to less understanding I Then we have the 
paradoKioal result that an increase In conceptuftl level im apparently 
associated with an increaee in understanding* This is Ml artifact of the 
eorfejition batvreen effort and level. Plagran 7 (a) Ahowi the positively 
correlated vectors 'effort' and * level *| (b) shows the Ufideratanding 
prajection on this planei and (c) shows the resolution of the effort 
vector into a component in the level direction wd a Mni|3onent orthogonal 
to the level direction* 
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LEVEL 



EFFOKT 



BTAmW 




(c) 

LEVEL A EFFORT ^ LEVEL 



UNDER- 
STANDING 



EFFORT 
^ LEVEL 



7. the multivariate analygi g of section ratings 

^agression analyaap ware carrtad out on U variable: 



1, 
2. 
3* 



UNDERSTANDING the mean rating for each motion on » understanding' 



TIME 
LINES 



4* SYMBOLS 

5. BEPCONCEPTS 

6, DIFCONCEPTS 

8 . EXBRCISEa 

9, DIAGRA^iS 
PBIOR 



10* 
11* 



LEVEL 



the mean rating for each section on Hlme' 
the number of lines of print in each section 
the number of lines of print with symbols in each section 
the number of tliaas mathematical concepts are mentioned 
in each section 

the number of diffarent mathematical CQncepts mentioned 
in each section 

the mean effort rating for concepts mentioned in each 
section 

the number of axerclses in each section 

the number of diagrams in each section 

the prior faniliarity of the cor^cept with lowest pridr 
familiarity in each section 

the level of the concept with the highest level in 
each section- 



First of all, I Ignored the data on introductions and sumnaries. Some 
oreU^lnary analyUs suggested that these short aectioM behaved verjr 
dlff^rentll fro/the ba»ie teaching sections and that th«lr presence 
^pJlSsed « B^n«ral .lengthV effect. Howver, their omission meant an 
even smaller sample sl^e of l4* 

In tha first analysis, I «sed all the variables 3 to " *° P^f*^* . 
undersJaidlng. in faci, only 'level' entered the equation aign.fiwntly. 

UNDERSTANDING = 4*8 - .11 LEVEL 

t 
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In the next analysis I used all the variables 3 to 13 to predict time. 
An examination of the T values for the regression eoefficients suggested 
that 'lines % 'diagrams », 'prior' and •lev'el* were the variables \*icli gave 
the greateat contribution to the predietlon. 

A stepwise regresaion was carried out using these four vaHables as 
predictors, »?Leval'« entered the aquation first, its correlation with time 
being 0.64, '^Diagrams" entered nrat, increasing the oiultlple correlation 
to 0,84, "Lines" entered next, giving R ^ 0.95* ''Prior" did not add 
aignlficantly to the equation. So the equation ist 



TIME 



-8,2 + 3 LEVEL + DIAQWHS + 0.08 LINES 

t ^ t 

T - 7.0 'x * 5.8 ' T « 4.4 



It is interesting to compare this equation w^ith the previous one. The 
variable 'level' is the moat powerful single predictor of both understanding 
and time. Hot^ever time is also dependent on •diagraras' and *lin^a«. I^hese 
latter two variables can be inteiisreted as measures of "length". Table 8 
shows that all measures of length (ie, varirtlea giving the number of times 
something occurs in the section) have higher eorrelations with time ^han 
with understanding t 



Table 8 Correlations of other variables with "understanding" and with "time" 
^ 'length' variables 

LINES SYMBOLS R^CONCEPTS DIFCONCMS EFTORT EXERClSEi DIAGRAMS PRIOR LEV 

UNDER- 02 ^23 -05 -JB ^51 -40 ^a? 27 «7t 

STANDING 

^^^^ 27 35 39 70 30 57 57 -40 6^ 

The resicjuals for the last equation are interesting i the sectioris on 
composition^ reverse and inverse mappings have greater times than predicted^ 
while the sections on specifying mappings, arithmetic of ftinctlons, and 
decomposition have lower times than predlated. These residuals suggest the 
effect of the unmeasured varlablei inherent dlffioulty (given conceptual level). 

8^ assessment questions 

Table 9 sets out the mean time for each of the assessment questions. 
The time for the TRA was 81 ninutes li^ile the time for the OTA was 43 minutes. 
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table 9 



TMA question 1 (assessed) 
TMA question 2 (non-asseaMed) 
CMA 





individual 




minutes 


questions 


ainuteg 


45 




:\ 


36 


2 


3. 4 J 


43 


3 






k 


« A 










6 


3.6 




? 


3.5 j 




VI 


3.9N 




2 






3 


3.6 




h 


4.6 J 




5 


5*3 



The above table also give^ the times for Indivldua OU itens. Items which 

™ed toae^ha? test .uch the -a„e «UiU. f ^ ^-,'Sa)'r'i«iS 
items do not vary much, ihe two shortest questions ^ "^f J^^^^^^^^^ 
recognising whether two simple functions were ono-one or JJ^^J^JHtf 
question involved the concept of «odi,l s function and of composition, (Vl). 



9. the validity of the ratlnfla 

Many people ask me why I u.e ratings as opposed to ^f^f^^ 
students^ LtLl performance. My answer foUo,.*. ^Tdo hot b.lieve 

perfomance items on one of the units later m the course. I do not believe 
that it is feasible to use only perforaance itmmi 

(1 ) Students have to spend a greater aaount of tine on performanee items 
as opposed to ratinga (cf. Tables 2 »nd 9). 

(2) So the cover of traditional OUs im lew *har, the cover of «y ratings. 

1 can ask rating question, about the prM«htatlon ""f^^' "^^^^ 
some doubtful reasoning need, to be appll#4 If we are to mm OIA 
results to diagnose faults in the presentMion, 

Often traditional CMAs teat a number of different sklllB at once - 
e" is the Item V5 time-conaumino bMWW of the ft»nctlon 
or baeause of eompositlQnt 

It is possible to construct easy or dlfflCTlt items «n the «sme skill 
- e.g. items Ul and U2 riqulred applytng *he one-one concept to very 
simple funetlons. 
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(3) 
ik) 
(5) 



My Investigations have already depionet rated to some a^tent the 
validity of tha ratings i 

(a) at the Mathematics Faculty Seminar, people's guespes ot the "top 3 
concepts'^ and the ''bottoia 3 concpats" ware 'right' atout 2 out of 
3 tiinesi 

(b) in sections 6 and 7^ I have demonstrated significant rtlationships 
between measurae of content and mean ratings; 

(c) the ratings for coneepta were eonalatent «th the ratings for sections 
- but theae two types of ratings ware given by two cottpletely separate 
groups of students I 

(d) graph k plots the facility index of OW itms (1974 i£ 1975) against 
the mean effort rating for that concept in the question requiring 
most effort* 
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10# conclusion 



This evaluation of Unit 1 has not been a full-scale ^'Detailed Evaluation 
I have asked only 158 questions as opposed to the 589 I am agklng about Unit 
Evan so I this report deiiionstrates the importarice of many of the features of 
Detailed Evaluatloni 

(a) the detail provided by information on individual eoncepts and sections 
enabled me to consider principles of presentation* This would not have 
been possible with inforraation at a grosser Iwel * Indeed, I felt a 
need for Infomation on speeifle examples and exarciaas* 

(b) the large nm^er of questions Biaant that this detail \mm available for 
a number of different objects (coneepts/aeetions) and for a number of 
dimensions (prior famillarltyi etc*)* By maklnQ the appropriate 
comparisons, I could r^ch iwpoFtant Interpretations of the information* 

(c) the content analysis of the unit made explicit one varTr Important 
variable (i*e. level) and enabl^ the testing of hypotheiSeB about the 
relationslhip between content and ratings^ 

(d) the multivariate analysis enabled me to select the most powerful 
predictors of varioua criterion variables - in doing so, to set up 
plausible models of stud^t learnings 

(e) although I have not interviewed any of the students, the request for 
comments on the questionnaire has given useful information about how 
the students have approached this task. 

(f) the report on the HathOTatlcs Faculty workshop has indicated how the 
results may be applied to the degign of new courses. (Af^emdiw 4-). 

Looking ahead, 1 mnt future valuations to have more questions about 
more detailed aspects of the unit. I would like to see relationships 
established between ratings and other content meaaurea besides » level'. 
Finally I would like to see wach stranger applleations of the results* 




Wh#n you have finished youi tudy of unit 1, please ecii^»let# this rating sh##t by 
putting a eirale round the appropriate number far of tiim thr## ^estlans and 

€aah of the ooneepts* 



a©ne©pts (page in t«t) 



set (4) 

9 m 

element (4) ^ 
equality of seta 

subset (5) ^ , ^ 

proper subset (5) ^ ^ 

mapping (12) ^ ^ ^ 

dCt*^. A (12) * p , 

cododK^ln (12) . ^ ^ 

iwage (12) ^ ^ ^ 

function (12) ^ ^ 

ordered pair (15) ^ ^ 

variable (16) # ^ ^ 

equality of mappings (17) ^ 

duflmy variable (18) ^ ^ 
constant function (18) 

graph (23) ^ ^ ^ 

function of two variablas (27) 

operator (29) ^ _ ^ 

difference (30) ^ ^ 

sua (30)^ ^ ^ . 

product (30) ^ ^ ^ 

quotient (30) ^ ^ ^ 
eosposition of Atnetlons (33) 

reverse mapping (38) ^ ^ 

one*one sapping (4l)^ ^ 
0n«--ttany aapping (41 )^ ^ 

iiany^ne laapplng (41 )^ ^ 

BSny-^sny sapping (41) . ^ 

inverse function (44) ^ 
Your resetions to this Mthod of fasdbs^i 
(I) V«»?« the iMt ructions <:lesr? 





prior 
familiarity 




effort 




current 
understanding 




.^fiata t al li 














/not at 








\_fwiiliar 






very 


iittlft 




\a 11 good 












.^familiar 






jery much 














4 








5 


1 


a 




4 


5 


i 


a 3 


4 


5 




1 


2 


3 


4 




1 


s 


3 


4 


5 


I 


a 3 






(1) 


1 


2 


3 


4 


5 


1 


2 


3 


4 


P 


1 


< J 


f 

4 




(2) ^ 


1 


2 


3 


4 


5 


I 


2 


3 


4 


5 


I 


a 3 


* 


5 


(3) ^ 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 3 




5 


(4) 


1 


2 


i 

4 


4 


5 


1 


2 


3 


4 


5 


I 


2 3 




5 


(5) 


1 


2 


3, 


4 


5 


1 


2 


3 


4 


5 


1 


2 3 




5 


(6) 


1 


2 


? 


4 


5 




2 


? 


4 


3 


1 




f 

4 




(7) 


1 


2 


3 


4 


5 




2 


3 


4 


5 


1 




1 

4 




(8) 


1 

1 1 


2 




4 


5 


1 


2 




4 




1 




4 




(9) 


! 1 


2 


3 


4 


5 


1 


2 


3 


4 


5 






4 


5 


(10) 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


I 


2 3 




5 


(11) 


: 1 


2 


3 


4 


5 r 


1 


2 


3 


4 


5 


1 


2 3 


4 


5 


(12) 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 




2 1 


4 


5 


(13) 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 3 


J 

4 


5 


(14) 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


a 3 


4 


5 


(15) 


i 


2 


3 


4 




1 


2 


3 


4 


5 


1 


2 3 


*# 


m 


(16) 


1 


a 


3 


4 




I 


i 


3 


4 


5 




2 3 






^ (17) 


I 


2 


3 


4 




1 


2 


3 


4 


5 


i 


2 3 


4 




(18) 


1 


2 


3 


4 




I 


2 


3 


4 


5 


1 


2 3 




5 


(19) 


I 










I 






4 


5 


1 


2 3 


4 




(20) 


1 


2 




4 




I 


2 




4 




I 


2 1 


4 


,5,.. 


(21) 


i 


2 


3 


4 


5 


I 


2 


? 


4 


P 


I 


2 2 


4 


^. 


<22) ,i 




1 


2 


3 


4 


s 


I 


2 


3 


4 


5 


I 


2 3 


4 


5 




1 

j 


1 


2 




4 


5 


1 


2 




4 


5 


1 




4 








































1 


2 




4 




1 


2 


,„3,„. 


4 


1 




4 






1 


2 


3 


4 


5 


1 


2 


3 


4, 




r'-l 




4 


5 






1 


2 




4 


.5,,. 


1 


2 












4 






1 


I 


e 


3 


4 


3 


1 


2 


3 


4 


? 






^ 


5 








3 


4 


5 


M 




3 


4 


5 


^1 




4 









m I 1 i 5 1 

(1) - (3) 



ERIC 



• I iiiiii If* 



S^ct ions^ Unit 1 

..... ...mplnto this «lu.ot by puttim, « circle round thc^ appropriate number 

H. y..ur un.lorwtftn.lin!! of thn n.-nt<rnl devolopmtMit of iUo«s m thn 

,,n.l l.v writing in ht,w many minut*^« you spent on .-fich section. You can 
,„.. V...U Htucly tlmoH as you work through thr- unit or you can juat try am! 

tliom aftof you hav#» romplotod your study. 



understanding of general 
j^velopniont of ideas^ 


time (miiiH 
(approx. ) 




not 
all 


at 

good 


J 


very A 
^good7 




' (pay*' in text) 












. _ — - . - 


,4S a whole s (^*) 


1 


2 


3 


4 


5 


1 i 1 

j ! ' 



Bee; t I on 1 . 0 
t. 1 
1.2 

section l.l.O 
1,1.1 
1,1,2 
1.1.3 
1-1.4 
1.1.5 
1.1.6 
1. 1,7 
1.2.0 
1,2.1 
1.2.2 
1.2.3 
1.2.4 
1,2,5 
1'.2,6 
1.2.7 



(p. viii) 
(p.!) 
(p. 30) 

(p.l) 
(p,l) 
(p,4) 
(p. 6) 
(p,li) 
(p. 14) 
(p. 19) 
(p. 27) 
(p. 30) 
(p. 30) 
(p,32) 
(p. 3?) 
(p. 38) 
(p. 40) 
(p. 44) 
(p. 47) 



(5) 
(6) 

(7) 

(B) 
(9) 
(10) 
(11) 
(12) 
(13) 
(14) 
(15) 
(16) 
(17) 
(18) 
(19) 
(20) 
(21) 
(23) 
(23) 



1 


a 


3 


4 


1 


2 


3 


4 


1 


a 


3 


4 



-4- 



2 
2 

2 

Si* 

2 
2 

a 

2 
2 
2 
2 
2 
2 
2 
2 
2 



3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 



4 
4 

4' 

4 
4 
4 
4 
4 
4 
4 
4 
4 
4 
4 
4 
4 



5 

5^ 

5 

5 

5 

5 

5 

5 



5 
5 
5 
5 
5 
5 
5 
5 



■4— 



I 







1 






^ i ^ 


1 


2 i 
2 ( 


3 i 


4 


rjir 


1 1 



(4'7) 



(a 

(1 1- 

(14^ 

(17 

(20 

(23 
(36 

(29 
(32 

(35 
(38 
(41 
(44 
(47 
(50 
(53 
(56 
(59 
(62 



■19) 

■22) 

•25) 
38) 

31) 
.34) 

=37) 
40) 
•43) 



-49) 



•55) 
58) 
.61) 
'6k) 



Suniniiary aiid conclusions (p4^) (24) 

Tick box at) if you noted your study times mm you went along O (it) 
Tick box m) if you Juat tried to estimate them after your study □ (4^) 

Your reactions to this method of feedbacks 

(1) W«re tho InstructlonB clear? 

(2) How long did it t«k« to fill in the rating aheet? ^ 

(3) Did you have any diffiqulty giving the ratings .. if so, ythmt were they? 



(63-67) 
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' Unit 1 assignment questionnaire 



*Try to give as accurate an answer to the questions as possible - 
but don't worry if you can't remeniber the exact dates (questions 
1 to 3) or the exact times (questions 10-15) - just give the best 
approKirnation I 

Day Month 

(1^3) 



!• I started my study of unit 1 on 

2, I posted TM^Ol to my tutor on 

3, I posted CMA41 to Walton Hall on 



(--6) 
(7^9) 



For each of questions 4 to 9 circle the option (A/B/C/D/E) which 
applies* 



4, I first read the CMA questions for unit 1 

5, I first afctempted the CMA questions for 

unit 1 

6* I completed the CMA questions for unit 1 
7* 1 first read the TMA question for unit 1 

8. I first attempted the TMA question for 

unit 1 

9. I Gompleted the TMA question for unit 1 



ABC 



A 


B 


C 


D 


E 


A 


B 


C 


D 


E 


A 


B 


c 


D 


E 


A 


B 


c 


D 


E 


A 


B 


c 


p 


E 


A 


B ' 


c 


D 


E 



Options 

A. before any serious study of the unit 1 tent 
B. during my first serious reading of the text^ 

C* after my first serious reading' 

D * during subsequent read ings' 

E* after subsequent readings 

Please indicate roughly how many minutes you spent on the following 
components of this unltr 



(10)^ 

(14) . 

(15) , 



10* T*V* broadcast and associated materials*-* 

11. Radio broadcast and associated materials.* 

12. Tutorial (excluding travel )•#*..*..* 

13* Correspondence text , • * 

14. CMA questions Ul .a*^**,. 

LJ2 CSS******..** m m m 



time (mlns) 



(18. 
(21- 
(24- 
(27. 
(30- 
(33- 



■20) 
■23) 
26) 
29) 
32) 
35) 



The Open University, 

Institute of Educational Technology, 

Walton Hall, 

Milton Keynes, 

MK; 6AA. 

Telephone: Milton Keynes 74066 



INSTITUTE OF 

THE OPEN UNIVERSITY educational technology 

7th Pebruary, I976 



Daor 



You may be aware that over the neart two years we are goino to rewrj.te the 
Mathematics Foundation Course. Obviously we shall need to molce a number o* 
aasumptions about how students leam from corrBspondenc© material ». In the 
past we have tested our aBauuptiona by asking "students" to fltudy draft material 
This method hag not been a coi^lete suceaBs. So we are trying an alternative 1 
afiking MlOO students like yourself about different aspects of their study. In 
this way we hope to establish some fluldino principles about how to design the 
materials for the new Foundation Course. 

A8 you may imagine, we already have a lot of Inforoatlon about Btudehtat 
assessment perfonnance and their overall reactions to the units. But we are 
still rather ignorant about studentB* reactions to more detailed aspects of the 
units. 

I vould like to ask you the following questions about your study of unit 1 
(Functions) 1 

(1) On vrtiat dates did yoU start and finish your work on unit 1? 

(2) At what stage did you work on the CHA and TMA questions? 

(3) How much time did you spend on the vaj-i-oue coinponents of unit 1? 

NOTEi Your answers to these questions will NOT affect your assesament 
grades in any wayi inforwatton about individual studenta will 
NOT be passed on to the Mathematics Faculty. 

I am asking only a limited nunber of students to answer theae particular 
questions. In this way 1 am cover many different aspects without askinfl any 
student to complete more than one questionnaire in the year, 

I would be very grateful if you would glva the answers to these que»tipn8 
on page 2. Please return this sheet in the reply-paid envelope. When 1 have 
analysed the results 1 shall send a aunaaary of ay conclualona to all thow who 
have taken part . 

Thank you for your co-operation. 

Yours sincerely 1 



jordon Burt 
Attaohaa t© th© Mathematlcp Wmmltf 



Appendix B 



^^Did you hava any difficulty giving the (concept) ratings if so, what 

were they? 



Having completed unit 1, I tended to base my Prior Familiarity rating 
on the difficulty I experienced. 

Prior Familiarity dubioUB^o alien notation. 

Having dona some of the work before (about 9 years ago), I had met many of 
the conGepts before, but had forgotten many of them. Therefore 1 could not 
say I was famiiiar with the concepta just before opening the book^ but oh 
meeting the concepts I found I knew aomething about thein# 

It's difficult to say how fainillar one was before with a concept, unless 
of course one knew nothing of it- 



Many of my problems in the text stem from adopting a different notation. 
It is therefore difficult to aseess farfniliarity as I had met moat concepts 
but often not in the same forma. 



The prior familiarity column posed a probrem. If you have never met the 
item before does it mean you are not at all familiar with th© concept? 
For instance the tenn proper subset was new to ma, but Iltnew about subsets. 
This same dllema was posed in No* 14, 12| etc. On the other hand did very 
familiar mean you had met and studied It before or ware vary familiar with 
the text of Unit 1 and the tanninology u/sed in it? I interpreted this 
column as very familiar if 1 had a good prior knowledge^ or had met similar 
work, and the other grades accordingly. 

The effort column was difficult to judge in isolation. I would have answered 
more assuredly if I was answering if the eoncept was grasped easily etc. 

The current understanding column was fairly straight forward* 

Assessing effort required. js*^ 

1 do not feel sure what my current understanding Is after only just 
Completing one unit* 

It is difficult to rate one's own understanding of a lituatlpn. fhe result ■ 
can only be judged by self escaml nation* 



One can only give the mtinge using the material (testt) read as eviden€# 
PR?r understanding- Further material (i«e^ QMeftioni} qouI^ ^evAlue one 

tKH^ Idea qt current utid^r|itfnding» C* 94. ^ 



-2- 

Very difficult xo be objective. Difficult to assess degree of knowledge 
required. Asaumed current knowledge to be coticamed with unit content. 

In some cases the idea was very quickly dealt with for example reverse 
mapping and inverse functions when I felt there was a lot of special cases. 
Perhaps a six-sense but sufficient to make one a little concemed about 
ones own ability. 
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Appendix 3 (cmH*) 



"Did you have any diffieulty giving the (section) ratings ««« if so, what 
were they?" 



Nol I took the average of each sectioni excluding the introductions and 
summa ry | fo r 1*1 and 1.2* 

3 ^ good? k - goodi 

Having done quite a lot of preparatory reading, familiarity with notione 
made it easier- 



Yes - I have done the woric in Unit 1 before so It was just revision 
therefore I am not represtentativep 

Over aimpli fication cauaed boredom, as with lack of explanation, lack of 
continuity- Deciding which region a rating was generated from^ 



I spent a lot of time going back in the unit and re--reading parts whose 
definitiona I was not clear about* 



Yes already worked through unit 1 two or three times difficult to 
recall time spent. 



Timing estimates very diffieult * kept picking up and re-reading. 

It would have been easier If the fortn had been racaivedi earlier so that 
(68) could be done. I presuma these results are analyaad by eomputer, 
could not bo%eB 4, 6 ^ 7 be programmed. 

Had gone through unit before racelvlng this form and so had to estimate, 
would have preferred to note Infomation while first studying unit* 
Consider estimates, done approx four weeks later, unrallabla* 

estimating study time at a later date. 

The data I am aupplying Is vary inaccurate, due to the length of time 
after study* I would hava preferred to tick box 68 • 



Not in ratings » but timings Imposjilbl^ to calculate since this quastionnaire 
was received after I had flnlahed the unit. 



Trying to think back aft#r auch a long tioe. Unit flrjft read in Deetmber, 



ioteffia! Memorandum 

from ^pdon Burt 



To MlOl Cours© t^m ©irf oth©rf) 

Subject A ra^rt on the Detailed Evaluatiofl workshops ^^^^ 2^*** March 1976 



The first workshop concentrated on the result® for th^ eoneapta in ^^.t 1, 
diseuBaing the interpretation of these results^ md their iDplieationB^ The 
second workshop studied each sub-section of the unit in turn, giving a critique 

of the presentation, and then relating the critique to the rasulte on that 
section and on the concepts in that section* Drawing on the discussions at 
these two woi^shopa, I have written the section "implications for MlOl" below. 
The section following that gives a brief account of the results (I can give the 
full report to anyone who wants it)o 

I think we should hold other Detailed Evaluation workshops *- In a number 
of guises. These are described in "suggested activities". Both workshops ran 
into time trouble! This might be avoided in ruture, if I give a quick overview 
of the results, using thra to point to specific problMS which we can then 
concentrate on (see the final section on "improved plan for Detailed Evaluation 
workshops") . 

ImplicatioM for M^O^ 

1, The presentation of conc^ts in the unlti 

* how many different eoncepts do you us© in the unitt 

m how familiar will the students be ^th each incept before they 

study the unit? 

d how many concept levels have you stacked on top of one another? 

• what is the irOierent difficulty of each wncept? 

s have you provided the right sort of ejcaaples j^nd CTe^cises to enable 
the student to understand and apply each concept? 

# are your eacplanations clear and simple? 

9 has your presentation become confused and ^^ged down in urtfielpful_ 

qual i f i ca t i ons^? 

9 have you structured the coneepts in the sections In a clear and 
simple manner? 

a is each concept essential to your alms * "If in doubt^ thraw It ©utl^ 

Unless we allocate some tine to taaaing these speoifle questions, they 
will tend to be ignored « 

a. How mich tiee do you thliUc the students ^11 spend on each section? 

Z shall provide an astiiMte based on mn analysis of your unit and the 
application of the statistical results for units 1, 2, and ?• you should 

have your o%m estimate too* ^ _ 
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The result g 

I have obtain^ tiquation^ illustrating the following r@latlmiahl|rji 

• the effort required to understand a coneapt im a funetion of how 
farniTi^^tiie students are with the eoncept before thoy start and 
oT^the^lerarehical level of the cancept# 

m studentB have less understanding of coneepts \4ilch are at a higher 
level and which have required greater cffor-t* 

m the average study time for each section im m function of th© 

conceptual level of each section and of the length of each 0action# 

Huch greater diocuasion centred on the tmt itself and on the reeulta for 
individual concepts* For example: 

- ® the concept "graph" was not Introduced till half war through the unit 
* despite the fact that it waa the concept which students were wost 
fi^llar with* 

& students were relatively familiar with the eoncept '^function" before 
studying the unit but had relatively little underatanding of the 
concept after the unit * we noticed that Vnit 1 (Functions) had no 
teaching about functions I 

m the concept "operator" required mucti effort and was little understood 
- yet this concept is In no way essential to unit 1# 

Su j ^ ^ested activities 

m MlQl Blocdcs which have overlap with HlOO units 3 to 7 might hold 
iiimxlar workshops to relate a tntual orlticlsm of these units to 
the Detailed Evaluation of these unitSa 

• MlOl Blocks i4iich have overlap with MlOO units f onwards might carry 
out a te%tual criticism of these units and use that as a basis for 
designing a Detail^ Evaluation of the relevant unit* 

m textual criticism workshop wight be run for HlOl units - addressing 
the questiorgon page !♦ 

m we might carry out Detailed Evaluation of those HIM units (e*g, at) 
which have implications for the rmaJce of the second-level courses* 

Improved plan for Detailed Evaluation woricshopS 

• quick five-minute presentation of general findings. 

e hand out results for individual concepts (or sections ete*) - for 

reference* 

• select just a few it^is for closer lamination. 

• carry out a t^tual crlticiwi on these few it^M. 
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